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A multi-model automated edu- 
cational and entertainment device 
(10) having improved functional vers- 
atility and which requires relational 
positioning of input data by the user 
which corresponds to required manu- 
al operations and procedures. The de- 
vice comprises a tablet (32) having a 
non-relational section (40, 42 or 44) 
for input of control or predefined in- 
formation and at least one relational 
section (56, 58, 60, 62 or 64) for input 
of data, a stylus (18) t or activating the 
non-relational and relational sections 
of the tablet (32), a microcomputer 
logic unit (82) for processing signals 
from the non-relational and relation- 
al sections of the tablet (32), and a 
memory unit (84) for storing pro- 
grams and data base for determining 
the accuracy or inaccuracy of the sig- 
nals provided the microcomputer log- 
ic unit (82). Visual and audio outputs 
(66, 16) can also be provided for op- 
erator reinforcement and a power 
emir /v% fnr activatina the device. 
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MULTI-MODAL EDUCATIONAL AND ENTERTAINMENT SYSTEM 

Technical Field 

Field of the Invention 

The present invention relates generally to the field of 
computerized teaching systems, and more particularly but not by 
5 way of limitation, to an improved portable multi-modal 
automated educational and entertainment system. 

Background Art 
Educational systems and devices have heretofore been 
proposed as teaching aids, such as for teaching multiplication 

10 tables and the like. The prior art systems and devices have 
generally been relatively simple with limited use capabilities 
and versatility, or extremely complex making it difficult for 
students and instructors to operate. 

One prior art device (U.S. Patent No. 3,009,262) with 

15 limited versatility comprises a light bulb carried on 

intersecting perpendicular rod members * the rod members being 
movable independently in two directions to locate the light 
bulb under an answer. The answer which overlies the positioned 
light bulb depends upon the distance the bulb is moved in 

20 perpendicular directions and indicates for example, the product 
of the numbers at their respective distances along the axes 
from the starting edges. 

Another prior art device, taught in U.S. Patent No. 
3,771,240, comprises a card having interconnected conductive ... 

25 elements over which a second card is positioned, the second 
card defining selected mathematical problems and possible 
solutions. Probes are used to select the problems and their 
proper solutions. Light bulbs are engaged upon choosing a 
correct solution. . - 

30 Still another prior art,: disclosed in U.S.. Patent No.. 

3,834,042, teaches a computational device for teaching 
mathematics by means of an apparatus for indicating correct and 
incorrect solutions to an equation having two independent 
variables for selected variable values. The apparatus includes 

35 a printed circuit board and electrically conductive strips, the 
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conductive strips forming a family of curves representing 
solutions to a desired equation of two independent variables. 
The family of curves is arranged in a manner so that the 
distance along one edge of the board represents the value of 
5 one of the independent variables, and the distance along 

another edge of the board represents the value of the other 
independent variable. The intersection of the lines parallel 
to the edges of the board intersecting over the curve repre- 
sents the solution to the equation of the two variables,. Each 

10 of the solution curves is connected to an electrically conduc- 
ting possible solution marker displayed on the top panel. A 
test probe is connected to a relay switch to energize the relay 
when the test probe is brought into contact with the proper 
solution member to which the voltage is conducted from the 

15 solution curve. The test probe contains a switch operable by 
pressure on the probe to engage a selected one of the two solu- 
tion indicating lamps r the incorrect lamp being ordinarily 
selected and the correct answer lamp being selected by positive 
potential being detected by the probe which the relay 

20 energizes. 

U.S. Patent No. 3,761,877 is representative of the more 
complex prior art systems and apparatuses r and is a system that 
utilizes a computer graphics input terminal comprising an 
optical or light position sensing device , a semi-transparent 

25 plate mounted in optical relationship to the sensing device , 
and a 1 ight-emitting~stylus movable by the operator over the 
plate surface against the sensing device. The sensing device 
provides an analog output corresponding to the position of the 
stylus on the surface of the sensing panel and an analog to 

30 digital converter provides a digital output for feeding 

positional data to computers and/or other digital or analog 
operated equipments This- typer of -system utilizes ~a~ central ize'd 
computer and is "not portable by its very nature. 

Another type of character recognition system, disclosed in 

35 U.S* Patent No. 4,184,147 , utilizes the movement of a stylus 
across a predesignated data pad comprising three or more 
electrically conductive areas. As the stylus traverses the 
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electrically conductive ares, a digital signal is produced 
which, when compared with a prestored table of characters, is 
either recognized or declared an input error. The signal is 
then transmitted to a computer or other device for processing. 
5 Another type of prior art character recognition system, as 

disclosed in U.S. Patent No. 3,835,453, provides for data input 
by a stylus that has a moving surface or ball, the stylus being 
constructed so as to produce a digital signal representitive of 
the movement of the surface or ball. The digital signal can be 

!0 transmitted either by wire or RF transmission to a computer or 
other similar device whereupon it is decoded or recognized as a 
valid character or declared an input error. 

Numerous other educational and character recognition 
systems and devices have been proposed. However, as with the 

15 before described prior art systems and devices, the other 

educational and character recognition systems and devices are 
inherently inflexible, substantially limited in function and 
versatility, or extremely complex in construction and 
operation. Thus, a need has long been recognized for automated 

20 teaching aids and/or entertainment systems which are portable, 
and yet have substantially unlimited versatility. Further, 
automated teaching aids are desired which require relational 
positioning of input data by the user substantially 
corresponding to manual operations and procedures in the 

25 solving of mathematical problems, spelling and the like. 

Summary of the Invention 
The present invention relates to a multi-modal automated 
educational and entertainment device having improved functional 
versatility and which requires relational positioning of input 

30 data by the user which corresponds to required manual 

operations and procedures. -Broadly, the present invention 
comprises a tablet having a non- relational section for input of 
control or predefined information and at least one relational 
section for input of data, a stylus for activating the 

35 non-relational and relational sections of the tablet, a 
microcomputer logic unit for processing signals from the 



WO 84/02995 



4 



PCT/US84/00129 



non-relational and relational sections of the tablet, a memory 
unit for storing programs and data base for determining the 
accuracy or inaccuracy of the signals provided the. 
microcomputer logic unit, visual and audio outputs for operator 
5 reinforcement and a power source for activating the device. In 
one aspect the present invention provides for a sequence of 
steps for operating the multi-modal automated educational and 
entertainment device under the control of stored programs. 
An object of the present invention is to provide an 

10 educational and entertainment device having improved 
functionality and versatility* 

Another object of the present invention is to provide an 
educational and entertainment device which requires relational 
positioning of input data by the user that corresponds to 

15 required manual operations and procedures in solving 
mathematical problems and the like. 

Still another object of the present invention is to 
provide a portable multi-modal automated educational and 
entertainment device suitable for use as a "stand-alone" unit 

20 or in combination with a master computer and a communication 
interface unit. 

Another object of the present invention is to automate 
manual operations required^dtn- a -classroom environment, thereby 
reducing the workload of a teacher in educating children. 

25 Other objects, advantages and features of the present 

invention will become apparent, to those skilled in the art from 
the following detailed description when read in conjunction 
with the drawings and appended claims. 



30 



Brief Description of the Drawings 
Figure 1 illustrates in diagrammatical form a plurality of 
multi-modal automated educational and entertainment devices ; 
constructed in accordance with the present invention, each of 
the devices connected to a master computer via a communications 
interface unit. 

35 Figure 2A is an isometric view of the multi-modal automated 

educational and entertainment device of the present invention, 
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and illustrating a stylus for activating data input pads 
forming relational and non-relational sections of the device. 

Figure 2B is a top plan view of the multi-modal automated 
educational and entertainment device 10. 
5 Figure 2C is a side plan view of the multi-modal automated 

educational and entertainment device 10 showing the location of 
the earphone jack 16A, the input/output jack 14A, on/off power 
switch 11, and the stylus. 

Figure 3 is a block diagram of the educational and 
10 entertainment device of Figure 2. 

Figure 4 is an enlarged perspective view of a data input 
pad, a plurality of which form the relational and 
non-relational sections of the device of Figure 2. 

Figure 5A is a top plan view of another embodiment of an 
15 educational and entertainment device constructed in accordance 
with the present invention, and illustrating a stylus for 
activating data, input pads forming a relational section and a 
plurality of non-relational sections of the device, the device 
adapted to receive an overlay member for selectively providing 
20 access to a predetermined portion of the relational section of 
the device. 

Figure 5B is a top plan view of an overlay member for use 

with the educational -and- entertainment device -of~Figure -5Av * • 

Figure 5C is an enlarged perspective view of the stylus 

25 232. 

t 

Figure . 6 is a logic -and . -process~flow^iagrAm»»of-j^ 
educational and entertainment devices illustrated in Figures 2 
and 5 . 

Figure 7 is a logic and process flow diagram of the "define 
30 function" of the diagram of Figure $. 

Figure 8 is a logic and process flow diagram of the "solve 
function" of the diagram of Figure 6. 

Figure 9 is a logic and process flow diagram of the "play 
function" of th diagram of Figure 6. 
35 Figure 10 is a logic and process flow diagram of the "show 

me function" of the diagram of Figure 6. 
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Description of the Invention 
Referring now to the drawings, and more particularly to 
Figure 1, a plurality of multi-modal automated educational and 
entertainment devices 10 (hereinafter referred to as 
5 multi-modal devices } are connected to a master computer 12 by 
any suitable means, such as an BS-232C interface protocol 
represented by lines 14. Each of the multi-model devices 10 is 
provided with a hearing device 16 and a stylus 18. The hearing 
device 16 can be any suitable unit known in the art such as a 

10 head phone, an ear plug, or a speaker. The stylus 18 can be a 
cognitive or non-cognitive device. The stylus 18 can be 
electrically connected to the multi-modal device 10 via line 
20, as shown, by RF signals, or the stylus 18 can be 
independent of the multi-modal device 10 as will be more fully 

15 described hereinafter. 

Referring now to Figure 2B, the multi-modal device 10 
. comprises a housing 30 having an " upper side 32. A plurality of 
input pads 34, 36 and 38 are disposed in predetermined portions 
of the upper side 32 of the housing 30 and form non-relational 

20 sections 40, 42 and 44, respectively. Similarly, a plurality 
of input pads 46, 48, 50, 52 and 54 are disposed on 
predetermined sections of the upper side 32 of the housing 30 
and form relational -sections* S6y 58~, 60 ,-62 'aid 64 , * ~ - ~ 
respectively. The term "non-relational sections" , as used 

25 herein, is to beunderstood to mean that there is no positional 
relationship-J^tween_the-JLnpufc.-pads^ 

sections", as used herein, is to be understood to mean that 
there is a positional relationship between the. input pads, such 
as would be true of the positions of digits in a mathematical 

30 problem, i.e. carrying digits, subtrahend, minuend and answer. 

The input pads 34 and 36 which define the non-relational 

sections 40 and 42 of the multi-modal" device 10,- upon - - 

activiation by contact with the stylus 18, provide selected 
control signals to the multi-modal device 10 and thus to the 

35 master computer 12 (shown in fig. 1); whereas the input pads 38 
which define the non-relational section 44 of the multi-modal 
device 10, upon activation by contact with the stylus 18, 
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provide cognition of predefined information to the multi-modal 
device 10 and thus to the master computer 12. Thus, the 
non-relational section 44 may be also referred to as a 
cognitive section. The visual marking, if any, accomplished by 
5 means of the stylus 18 on the input pads 34,36 and 38 of the 
non-relational sections 40, 42 and 44 is inconsequential to the 
operation of the multi-modal device 10. The input pads 46, 48, 
50, 52 and 54 defining the relational sections 56, 58, 60, 62 
and 64 of the multi-modal device 10 are specifically used for 

10 the marking of symbols or information by means of the stylus 

18. When the stylus 18 is brought into marking engagement with 
any of the input pads 46, 48, 50, 52 and 54 defining the 
relational sections 56, 58, 60, 62 and 64 of the multi-modal 
device 10, the activation of the input pad provides location 

15 indication signals to the multi-modal device 10, while the 

marking is indicative of the information associated with the 
particular input pad location. •• - - 

The input pads 34, 36 and 38 in the non-relational 
sections 40, 42 and 44, as well as the input pads 46, 48, 50, 

20 52 and 54 of the relational sections 56, 58, 60, 62 and 64 are 
shown as being made up of two half pads, each half-pad being 
numbered. All upper half pads bearing the same number are 
.. --electrically connected •together- wdrthin- the-^i ti-modal- device-- =• 
10 and may be viewed in a logical sense as comprising rows. In 

25 the embodiment shown, there are therefore 15 rows numbered" 1 
.... through 15. Likewise,- all-lower- < halJr^ds J J3ear»i^~.AheL.sa m e ,r^» 
number are connected together within the multi-modal device 10 
and may be viewed in a logical sense as comprising columns. In 
the embodiment shown, there are therefore 16 columns numbered 1 

30 through 16. When the stylus 18 is brought into contacting 

engagement with an upper half of. . an., activation - pad,- it provides 
. an input impulse to^fehe logic- J*nit» &2*on the. row- of activation = 
and when the stylus 18 is brought into contacting engagement 
with the lower half of the activation pad, it provides an input 

35 impulse to the logic unit 82 on the column of activation. 
Alternatively, the stylus being brought into straddling 
engagement with the upper and lower halves of any activation 
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pad provides input impulse to the logic, unit 82 on the row and 
column of activation. The logic unit 82 through its stored 
program is then able to identify th specific pad location of 
activation by the stylus 18 from the row and column input 
5 impulses received. 

The multi-modal device 10 further comprises a visual 
display unit 66 and a voice unit 68 (as shown in Figure 3) 
which are used for positive and/or negative reinforcement to 
the operator of the multi-modal device 10. The visual display 

10 unit 66 comprises a plurality of lights and alphanumeric LEDs, 
LCDs or other display means well known in the art which will be 
described in more detail hereinafter. 

In the operation of the multi-modal device 10 , depending 
upon its design, certain information may not be input into the 

15 logic unit 82 via any of the input pads, such information none 
the less being important in the operation of the device. Pro- 
vision by way of a scratch pad 120 is made for marking - informa- 
tion, using the stylus 18, in order to facilitate the operator 
in remembering this information. The scratch pad 120 can be 

20 provided on the upper side 32 of the housing 30 of the multi- 
modal device 10. The scribing surface of the scratch pad 120 
can be any suitable means which is capable of being marked upon 
with the stylus 18, when 'the stilus 18 is capable of producing" 
a mark, such as a carbon core stylus or a stylus having a felt 

25 tip type pen. Further, the scribing surface of the scratch pad 
120 should be rapablg- :c€--I^ 

the multi-modal device 10 when it is desired to remove any 
markings from the scribing surface of the scratch pad 120. 

In a cognitive system wherein impulses from the stylus 18 

30 are deciphered by the multi-modal device 10, misrecognition 

creates certain practical difficulties.- In such an event, the 
indicia needs to -be remarked^wittr no guarant^esTthat: it will be 
properly recognized the next time. It is conceivable that 
while the above misrecognition was in progress, marking on 

35 another pad location had commenced leading to confusion as to 
how to operate th d vice in an orderly fashion. In one 
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method, after completing the marking on a pad location, the 
operator pauses for a certain length of time until the system 
recognizes the written indicia and provides a visual or sound 
feedback to the operator indicative of a recognized input. In 
5 another case, after completing the marking on a pad location, 
the operator activates an activate pad 70 with the stylus to 
indicate completion of marking on any given relational pad. In 
either case, if the system misrecognizes the indicia, then 
instead of rewriting the indicia, the operator simply uses the 
10 stylus to activate the appropriate non-relational data input 
pad to "fix" the error and moves on to the next operation. 

Referring now to Figure 3, a block diagram of the 
multi-modal device 10 is illustrated. The multi-modal device 

10 , in addition to the input pads defining the relational and 
15 non-relational sections, and the stylus 18 as heretofore 

described, comprises a power source 80, a logic unit 82 and a 
memory unit -84..- The power source 80 employed- to electrically - 
activate the multi-modal device 10 can be an internally 
disposed battery, such as a standard 9 volt rechargable 

20 battery, or the power source 80 may be a conventional 

receptacle for connection to an external power source. When 
employing an internally disposed battery as the power source 80 
: it is desirable- to provide a -step- down -transformer/converter -so 
that the battery may be recharged when required. 

25 When power is turned on, the signals received by the logic 

.. ...unit. -82 via buss -86 is. the result. of-_ac£ivation.-of- one- or_more - 
pads on the upper side of tablet 32 by the stylus 18 in. a 
pre-determined sequence. The logic unit 82 can be any suitable 
unit capable of carrying out the desired functions, such as a 

30 microprocessor. The logic unit 82 will process either control 
signals, location indication signals or data signals in 

. accordance -with its stored- programs. .as will be. more .fully. ^ 

described hereinafter with reference to Figures 7, 8, 9, 10 and 

11. Typical of a logic unit which may be employed in the 
35 multi-modal device 10 of the present invention is a 

microprocessor model 6502 CPU manufactured by Rockwell 
International, Inc. 



WO 84/02995 



-10- 



PCT/US84/00129 



The logic unit 82 is connected to the upper side (tablet) 
30 via a bi-directional buss 86; 'and the logic unit 82 is 
connected to the stylus unit 18 via a bi-directional buss 20 . 
The logic unit 82 is capable of controlling the voltage levels 
5 in the bi-directional buss 86 and the bi-directional buss 20. 
By first setting pre-detennined voltage levels on the bus 86 
and/or the buss 20, and then sensing the voltage levels on the 
buss 86 and/or the buss 20, the logic unit 82 is able to 
decipher the operation of the upper side (tablet) 30 and the 

10 stylus unit 18 . The impulses sensed by the logic unit 82 
through the buss 86 provide location of pad activation 
information* to the logic unit 82 while the impulses sensed by 
the logic unit 82 through the buss 20 provide the logic unit 82 
with information on the location of pad activation and/or the 

15 symbol being marked on the input pad* 

The memory unit 84 of the multi-modal device 10 is 
connected to the logic unit 82 via buss 88* The memory unit 84 
stores programs, data base and temporary scratch pad memory, 
either singularly or in a plurality of non- volatile read/write* 

20 memory, such as bubble memory, or volatile RAM (Random Access 
Memory) and/or ROM (Read Only Memory). Thus, the memory unit 
84 provides an information signal to the logic unit 82 to 
compare with the input impulses received by the logic unit 82 
from the activation of the relational and non-relational input 

25 pads of the multi-modal device 10. 

The voice -jinit. 68 is ^operably -connected t0*1±^ 
82. The voice unit 68 may include a voice synthesizer 69 
connected to the logic unit 82 via a buss 92. An output 
signal, representative of process data in the logic unit 82, is 

30 directed to the voice synthesizer 69 of the voice unit 68 via 
the buss 92, the process data being in the form of a signal 
used for both positive and negative reinforcement of the 
operator, as well as for prompting the us r, such as in the 
presentation of a sp lling word and the like. Any suitable 

35 voice synthesizer well known in the art can be employed. The 
output of the voice synthesizer 69 can be delivered to the 
operator through a headset/earphone 16 via line 96, junction 
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98, and line 100. Alternatively, the output signal of the 
voice synthesizer 69 may be delivered to a speaker 102 via line 
96, junction 98, and line 104. In addition to the voice 
synthesizer 69, the voice unit 68 comprises a tone generator 
5 106. The tone generator 106 is connected to the logic unit 82 
via a buss 108. The tone generator 106 generates a tone or 
tune which functions as positive and negative reinforcements 
for the operator of the multi-modal device 10. The output of 
the tone generator 106 can be delivered to the headset/earphone 

10 16, via line 110, junction 112, and line 116, or the speaker 
102 via line 110, junction 112, and line 114. 

The visual display unit 66 of the multi-modal device 10 is 
connected to the logic unit 82 via a buss 118. The visual 
display unit 66 comprises a plurality of lights and 

15 alphanumeric LEDs or LCDs (shown in Figure 2), or suitable 

means which provide a visual readout of the signal transmitted 
to the visual- display unit 66 from the logic unit 82. - Thus> " ' " 
under program control, the logic unit 82 can produce the 
following: an output visual signal via buss 118 and the visual 

20 display unit 66; musical and other tone sounds via the buss 
108, the tone generator 106, line 110, junction 112, and line 
114 to the speaker 102, or via the buss 108, the tone generator 
106, line 110, junction 112, and .-l-ine-116* to- the- • — 
headset/earphone 16; speech or vocal sounds via the buss 92, 

25 the voice synthesizer 69, line 96, junction 98, and line 104 to 
the speaker 102, or via the buss~92, the-vaice..synthesizer..69i r - r « 
the line 96, junction 98, and the line 100 to the 
headset/earphone 16. As previously stated, the visual display, 
the musical or other tone sounds, and the speech or vocal 

30 instructions can be either a positive or negative reinforcement 
to the operator of the multi-modal device 10, such depending 
upon the correlation .between -.the.-data^input.~into.. . the — — — 
multi-modal device 10 and the program data contained in the 
memory unit 84 of the multi-modal device 10. 

35 The multi-modal device 10 as described above is a 

self-contained portable educational and entertainment system 
capable of enabling the operator, such as a child, to carry out 
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preprogramed functions. However , the capabilities of the 
multi-modal device 10 can be substantially increased , while 
still maintaining an individual unit even when a plurality of 
such units are used by a number of different operators, by 
5 interconnecting the plurality of the multi-modal devices 10 to 
the master or central computer 12 as illustrated in Figure 1. 
In such instance , the logic units 82 of each of the multi-modal 
devices 10 has an integrated serial and/or parallel I/O 
(input/output) port 124 connected to port 126 of the master 

10 computer 12 via a buss 14. The interconnection of the I/O port 
124 of the logic unit 82 and the I/O port 126 of the master 
computer 12 allows transmission and receipt of control and 
input/output data in and from the master computer 12 to the 
logic unit 82 of each of the multi-modal devices 10 and 

15 vice- versa. 

As shown in Figure 2, the input pads 34, "35 and 38 of the~ 
respective non-relational sections 40 , 42 and 44 f the input 
pads 46, 48, 50, 52 and 54 of the respective relational 
sections 56, 58, 60, 62 and 64, and the activate pad 70 in the 

20 non-relational section 40 of the multi-modal device 10 can be 
of any suitable construction. Further, such pads can be 
constructed to be 'activated b^ pr^ssurev light, heat or* any 
other means known to the. art. For example, if the input pads 
are activated by pressure the stylus 18 may be an ordinary 

25. pencil or pen which. -is^nofc physically -,Gonnected^o^the chousing- — ~- 
30 of the multi-modal device 10. The relational and 
non-relational input pads, physically located on the upper 
surface 32 of the body portion 30, may be constructed variously 
such that they may be no more than demarcations, with switches, 

30 sensors or other means known to the art underneath them for 
detection of activation.- -Thus> ttoe-4*nput^datari»-^e , i 
unit 82 via the input pads for the selection of the desired 
functions, such as control, cognition input, display or the 
like, can be accomplished by either using the stylus or one of 

35 the operator f s fingers to apply pressure to the required input 
pad in the pred termined non-relational section of the 



WO 84/02995 _ 13 _ PCT/US84/00129 

multi-modal device 10, while the pressure of marking indicia on 
the relational pads provide location indication impulses to the 
logic unit 82. 

While activation of the various pads of the multi-modal 
5 device 10 can be accomplished in several ways as described 
above, and the pads can be fabricated of various types of 
materials, configurations and designs, desirable results have 
been obtained when the input, pads of the multi-modal device 10 
are constructed as illustrated in Figure 4. Since each of the 

10 input pads, i.e. the input pads forming each of the relational 
sections, non-relational sections, and the activate pad 70, are 
substantially identical in construction, only the activate pad 
70 will be described in detail with reference to Figure 4 . 

The activate pad 70 comprises a body portion 130 having an 

15 upper surface 132 and an opposed lower surface 134. The body 
portion 130, depicted as having a substantially circular 
configuration > is characterized -as- having- a-f irst 'half -136~and^ 
a second half 138. The body portion 130 is constructed of an 
electrically conductive material and an insulating medium 140, 

20 such as plastic, is disposed therein so that the first half 136 
of the body portion 130 is electrically insulated from the 
second half 138 of the body portion 130. An electrically 
conductive lead or pin 142 -is secured the --fixs1r~hal-f-13f6 ~o£ - 
the body portion 130 so as to extend from the lower surface 134 

25 of the body portion 130 substantially as shown. Similarly, an 
el ectrical ly conductive. *lead ©c.pia 144 -is. .seGured-^a -the — ■. — 
second half 138 of the body portion 130 so as to extend from 
the lower surface 134 of the body portion 130 substantially as 
shown. Thus, in an assembled position of the activate pad 70 

30 with the housing 30 of the multi-modal device 10 the leads or 
pins 142, 144 contact the jelectrical .circuit contained within 
the housing 30 of ^the multi^iwsdal^evice. la .such that 
electrical signals can be generated when the input pads, such 
as the activate pad 70, are activated by movement of the stylus 

35 18 across the upper surface 132 of the body portion 130 

heretofore characterized as the first and second portions 136 , 
138. 



WO 84/02995 



14- 



PCT/US84/00129 



Any suitable electrically conductive material can be 
employed to fabricate the body portion 130 of the activate pad 
70 and each of the input pads forming the r lational and 
non-relational sections of the multi-modal device 10* Such 
5 materials are well known in the art and thus a further 

explanation of such materials is not believed to be necessary 
herein. Further, the upper surface 132 of the body portion 130 
of the pads forming the non-relational sections 40 , 42, and 44, 
as well as the activate pad 70 may be treated or finished such 

10 that marking by the stylus is minimized or inhibited* However, 
since indicia are marked on the pads forming the relational 
sections 56, 58, 60, 62 and 64, i.e*, input pads 46, 48, 50, 52 
and 54, respectively, the upper surface 132 of each of the 
input pads may be formed of a material for writing on with a 

15 carbon base tip or an ink bearing electrically conducting fiber 
tip* However, in such instances the upper surface 132 of each 
of the body portions- of* the input -pads must be~fabrieated- s to * 
facilitate being marked on as well as being capable of the 
operator erasing or removing the markings once the use for such 

20 markings as a means for visual recognition has ceased . It 
should be noted that while the activate pad 70 has been 
illustrated as having a circular configuration, the particular 
configuration- of the activate- pad -?0> • and -each -of—the -other - — 
input pads of the relational and non-relational sections of the 

25 multi-modal device 10 is not critical and can vary depending 
upon the particular asthetic qualities desired. in. the overall 
appearance of the input pads and their relationship with the 
housing 30 of the multi-modal device 10, 

The stylus 18 employed to activate the input pads of the 

30 multi-modal device 10 can be a non-cognitive unit or a 

cognitive unit* When employing a non-cognitive unit as the 
stylus 18 (as illustrated in Figures 1, 2, 3 .and 5), the 
non-cognitive unit can be a carbon core pencil- like member, 
which may be electrically connected to the logic unit 82 via 

35 line 20 and the housing 30 of the multi-modal device 10 . 

Optionally, the carbon core pencil-like member may be a stand 
alone unit, when it makes straddling engagement with the upper 
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and lower halves of any pad. On the other hand, when employing 
a cognitive unit as the stylus 18, the cognitive unit may be 
constructed so as to provide indicia indicating signals to the 
logic unit 82 of the multi-modal device 10 via either an 
5 electrically conductive line or buss (represented by line 20 in 
Figure 1), or RF signals. Any suitable cognitive unit can be 
employed as the stylus 18 of the multi-modal device 10 .Typical 
of such a cognitive unit is the code generating and receiving 
apparatus disclosed in my U.S. Patent No; 3,835,453. 

10 The code generating apparatus of U.S. Patent No. 

3,835,453, the disclosure of which is hereby expressly 
incorporated by reference, is constructed such that a writing 
or marking can be made on the upper surface 132 of the 
relational input pads of the multi-modal device 10 so as to 

15 provide the logic unit 82 with input pulses indicative of the 
location where the marking was accomplished; and the apparatus 
simultaneously generates, and- transmits indicia indicating- 
signals to the logic unit 82 of the multi-modal device 10 . The 
indicia indicating signals transmitted to the logic unit 82 are 

20 in the form of an indicia code representing the marked indicia. 
In the use of prior art systems employing a cognitive 
unit, problems have been encountered in that the system may not 
recognize the character made by the cognitive unit. -Upon -- 
completion of marking on an input pad of a relational section, 

25 the activate pad 70 may be activated, thereby providing input 

impulses to. the. logic unit. 32 that recognition of -the. indicia ~ 
code may be started. In the event of non- recognition of the 
indicia indicating signals from the code generating apparatus 
by the logic unit 82 of the multi-modal device 10, the operator 

30 contacts the code generating apparatus with the appropriate 

input pad of a predetermined non-relational section so that 

, data represented hy the activation of the input pad. J.s_ entered--- 
into the logic: unit 82 to correspond to the marked indicia 
which was misrecognized. 

35 Referring now to Figure 5 A, a second embodiment of a 

multi-modal device 210 is illustrated. The multi-modal device 
210 comprises a housing 212 having an upper side 214. A 
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plurality of non-relational input pads 216, 218 and 220 are 
disposed in predetermined portions of the upper side 214 of the 
housing 212 and form non-relational sections 222 , 224 and 226 , 
respectively. Similarly, a plurality of relational input pads 
5 228 are disposed on the upper side 214 of the housing 212 and 
form a relational section 230. The non-relational input pads 
216, 218 and 220, which define the non-relational sections 222, 
224 and 226 of the multi-modal device 210/ upon activation by 
contact with a stylus 232, provide selected control and data 

10 input signals to the logic unit 82 of the multi-modal device 
210 and thus to a master computer 12 (shown in Figure 1)? 
whereas the relational input pads 228 which define the 
relational section 230 of the multi-modal device 210, upon 
activation fay marking engagement of the stylus 232, provide 

15 location data to the logic unit 82 of the multi-modal device 
210 and thus to the master computer 12 . The input pads 228 of 
the relational section 230 are interconnected in rows and 
columns such that location data input is transmitted to the 
logic unit 82 in the multi-modal device 210 for processing as 

20 heretofore described with reference to the multi-modal device 
10. 

The stylus 232 , shown in figure 5C, includes an elongated 
housing 234 which may be gripped by the human fingers and a 
centrally disposed core 236 having a first end 238 and an 

25 opposed second end 240. The stylus 232 can be an independent 

element, or the stylus 232 can -be electrically ^coimected ta the- 
logic unit 82 of the multi-modal device 210. For example , an 
electrically conductive line or element 242 can be employed to 
interconnect the second end 240 of the core 236 of the stylus 

30 232 to the logic unit 82 via *th& housing 212 of the multi-modal 
device 210 so that an electrical potential can be established 
therebetween. 

The first end 238 of the core 236 extends outwardly from 
the elongated housing 234 of the stylus 232 and forms a tip 
35 portion* Th core 236 can be formed of a material capable of 
making a visible marking on the relational input pads 
contacted, or may be constructed of a non-marking materials A 
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typical example of a material suitable for use as the core 236 
of the stylus 232, which is capable of making a visible mark on 
the relational input pads, is carbon; whereas an example of an 
non-marking material which can be employed as the core 236 of 
5 the stylus 232 is a steel rod. 

The multi-modal device 210 further comprises a visual 
display 244, a speaker 246 and an activate pad 248. The visual 
display 244 and the speaker 246 are employed to provide visual 
and sound signals for positive and/or negative reinforcement to 

10 the operator of the multi-modal device 210 in the same manner 
heretofore described with reference to the multi-modal device 
10. The activate pad 248, which upon activation produces a 
signal to the logic unit 82 contained in the housing 212 of the 
multi-modal device 210 so that the logic unit 82 can receive 

15 and recognize the stylus signals, is similar in construction 
and function to the activate pad 70 of the multi-modal device 
10 heretofore- described in-de tail;- -Therefore, -no- further 
description of the visual display 244, the speaker 246 or the 
activate pad 248 is believed necessary. 

20 The multi-modal device 210, in addition to the logic unit 

82 further comprises a power source 80 and a memory unit 84, 
each of which is identical in construction and function to the 

... . power source 80, the -iogic-uniir 82 and- the memory- unitr«4 of— ~ 
the multi-modal device 10 heretofore described in detail with 

25 reference to Figure 3. Therefore, a detailed description of 
such <mponents^wlth^ference^^ 

would be redundant and is not believed necessary for one to 
fully understand the construction and operation of ther 
multi-modal device 210. 

30 Referring now to Figure 5A and 5B, the multi-modal device 

210 further comprises -a plurality of- overlay -members -25 0-r- (only 
one being illustrated*4n .Figure. ^ 5B^«each-e€^^eKovera^ members 
250 having predetermined openings 252 therein so that a 
selected portion of the input pads 228 defining the relational 

35 section 230 of the multi-modal device 210 can be selectively 
exposed to the operator via the openings 252 while the 
remaining input pads 228 are covered by the overlay member 250 
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and thus not available to the operator as long as the overlay 
memb r 250 is maintained in position on the upper side 214 of 
the housing 212. For example, the overlay member 250 depicted 
in Figure 5B is provided with a plurality of openings 252 in an 
5 appropriate location so that when the overlay member 250 is 
positioned on the upper side 214 of the housing 212 the input 
pads 228 of the relational section 230 defining an "add" 
function of the multi-modal device 210 are positioned within 
the opening 252. By the use of a different mask, the same 

10 relational pads may be used for performing other functions, the 
mask providing the window for selectively exposing and bringing 
into relational positioning the "free form" pads for 
accomplishing the desired function. While the overlay member 
250 has been illustrated as containing a plurality of openings 

15 252, it is readily apparent that the overlay member 250 can be 
provided a single opening of a predetermined size and shape 
required to expose the preselected input pads 228 of the 
relational section 230 defining a particular function. 

The overlay member 250 can be formed of any suitable 

20 material. However, it is preferable that the overlay member 
250 be formed of a thin sheet of durable, flexible material so 
that the overlay member 250 can be used a number of times. 
Thus, desirably the overlay material will be formed of a 
plastic or heavy duty paper material. 

25 Numerous methods can be employed to secure the overlay 

member 250~in a -stabie-position 'onr-the* upper-Bide-^i^of-the-— 
housing 212 to insure that the proper input pads 228 are 
exposed via the openings 252 formed in the overlay member 250. 
The particular method employed will be dependent, to a large 

30 degree, upon whether the input -pads 228 of the relational. 

section 230 are substantially flush with the upper side 214 of 
the housing 212, as shown in Figure 5A, or are in a raised 
position above the upper side 214. As shown in Figure 5A, 
ridge assembly 254 is disposed on the upper side 214 of the 

35 housing 212 so that the ridge assembly 254 encompasses the 
input pads 228 of the relational section 230* The ridge 
assembly 254 can be a continuous segment r as shown, -or can be a 
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plural ity of intermittant segments disposed around the 
periphery of the relational section 230. 

The process of implementing the multi-modal devices 10 and 
210 are substantially identical. Thus, the process of 
5 implementing the multi-modal device 10 will be described in 
more detail with reference to Figures 6-10 of the drawings. 

Figure 6 illustrates a process 300 for selecting the 
various functions that are available on the multi-modal device 
10. in carrying out the process a sequence of steps and 

10 decisions are required, the first step requires a decision of 
whether or not to transmit or receive data or control signals 
from the host or master computer as shown in block 302. If a 
"yes" decision is made at this particular point, block 304 is 
invoked which will either allow the transmission or receipt of 

15 data and/or control signals to/frcm the host computer by means 
previously described herein. If a "no" decision is made, one 
of the four functions represented by block 306 is selected by 
activation of the appropriate input pads. The particular 
selection may be made to proceed to a "define", "Solve", "play" 

20 or "show me" mode. The next decision is to determine whether 
or not to "reset", in the event that an erroneous signal has 
been entered. The "reset" is represented by block 308. If 
"reset" is selected a signal is fed to connector "A", 
represented by block 310, and the selection process is 

25 restarted from the input connector rt A". On the other hand, if 
the reset is not activated- a signal' ±s~d±rec^d^to hldcJr-312 
for a determination as to the validity of the selection. If a 
valid selection has not been made, a signal representing an 
error message would be produced in block 314 and a signal 

30 directed to connector point "A", block 310, to restart the 

procedure* If a "correct" selection has been made a signal is 
directed to one of four possible connector "B", "C", "D" or 
"E^r represented by the blocks 316, 318, 320 and 322, 
respectively, whereupon the signal would proceed to one of the 

35 following blocks, namely t a "define* block 324, a "solve" 
block 326, a- "play" block 328, or a "show me" block 330. 

Figure 7 illustrates the "define" function of Figure 6 and 
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the procedures involved therein* The "define" function is 
accomplished via signal input through connector "B" represented 
by the block 316 • The "define" function is represented by the 
block 324 and in operation the operator decides what to do 
5 within that function,. Block 327 provides for the selection of 
the following; 1) math problems , 2) spelling, or 3) music. 
Upon selection of one of the above functions the process allows 
the operator one of two options at block 329, either to "reset" 
or to proceed to the next part of the process. In the event of 

10 "reset" is selected , a signal is directed to connector "A", 
block 310. If not "reset", a signal proceeds to a valid 
selection block represented by the numeral 313 where a 
determination is made whether the signal represents a valid 
selection. If the signal represents an incorrect, selection the 

15 operator is given an error message represented by block 315, 
and the signal would proceed to connector "B" , block 316 and 
the process would be repeated. If the signal represents a 
valid selection the signal proceeds to one of three connectors 
"F% "G", or "H", represented by the blocks 331, 332 and 334, 

20 respectively. These blocks allow the process then to continue, 
in the case of connector "F", block 331, to mathematical 
operations represented by block 336; in the case of connector 
"G", block 332 to spelling operations represented by block 338; 
or in the case of connector "H", block 334, to music operations 

25 represented by-block- 340^- OJpon- selection -=of none- of- -the* t&reer* 
possibilities, i.e. math, spelling or music, an option to reset 
is provided represented by block 342 to allow for reselection 
if needed. If the reset function is not involved, the process 
proceeds to block 344 for data input and then onto block 346, a 

30 valid entry decision block. If an invalid entry was made, the 
signal would continue to block 348 whereupon an error message 
would be produced and the signal would proceed back to the 
connector "P", "G", or "H" depending upon where the process 
originated from, if a valid entry was made, the signal would 

35 continue onto block 350 where a determination would be made as 
to whether or not a definition had been completed. If the 
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definition had not been completed, additional input o£ data is 
entered by repeating the process through block 344. If the 
definition had been completed , the process would store the 
definition of the problem in the block represented by the 
5 numeral 352. Upon storage of the definition of the problem, 
the process would be repeated through whichever connector the 
process started until the -definition process is completed. As 
detailed hereinabove, connector "F" starts an example of a 
mathematical problem definition process. The procedure would 

10 be followed in the same manner in each of the three blocks 
whether it was connector "F" for math, connector "G" for 
spelling, or connector "H" for music. 

The solve function flow process of the multi-modal device 
10 is illustrated in Figure 8. An input signal is received 

15 through connector "C", represented by block 318, and the signal 
is directed to solve block 326. In this instance one of five 
possible functions represented by block 358, namely, add, 
subtract, multiply, divide or spelling may be selected. Upon 
selection of one of the five functions, an opportunity is 

20 provided to "reset" if an error has been made. The "reset" is 
represented by block 360. When it is determined that an input 
error has been made, and the reset activated, a signal is 
directed to the connector "A" represented by the block 310. If 
the input is proper, the signal is continued to block 362 for 

25 determination of whether the signal represents a valid 

sel ection . Two;-cccurances^can-.happ^ 

selection was invalid, a signal is produced to generate an 
error message, represented by block 364, and the signal would 
proceed through connector "C", i.e., block 318. On the other 

30 hand, if the selection was correct, a signal is provided to one 
of five selected functions, namely add, substract, multiply, 
divide or spelling, represented by the" blocks 366/^368; "370T '"' 
372 and 374 via connectors "I", "J", "K", "L" or "M", 
represented by blocks 376, 378, 380, 382 and 384, respectively. 

35 Upon selection of any one of these five functions a 

determination as to whether or not a "reset" is necessary, and 
in the event of an input rror, the selection can be reset via 
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reset block 386,. If ■'reset*, a signal is passed to connector 
"C", represented by the block 318 so that another selection 
could be made* For example , if doing an add function and the 
subtract function was input, the process could be "reset" and 
5 the correct add function input. If the correct function has 
been input, and the process not "reset" , a signal is directed 
to block 388 for input of data; and then into block 390 where a 
determination of the validity of the entry is made* If the 
entry is invalid, a signal is directed to block 392 where an 

10 error message is produced, and the signal representing the 

error message is directed back to block 388 so that new data 
can be input. The signal representing the new input data would 
again be directed to block 390 for determination as to whether 
or not a valid entry had been made. In the event that an entry 

15 was valid, a signal representing the valid entry would continue . 
to block 394 where a determination would be made as to whether 
or not the solution was complete. If the solution is not 
complete, in that the solution required more steps, a signal 
would be passed back to block 388 for input of additional data. 

20 if the solution is completed, a signal is generated which is 

passed to block 396 where the signal representing the answer is 
evaluated with stored data to determine whether or not it was 
correct. The process would continue by providing. a signal from 
block 396 to the connector from which the process originated, 

25 such as the connector "I" if the "add" function being 

processed . The -above described' -process -could - be-followed for — ^ -r.- 
any one of the five possible functions, namely add, subtract, 
multiply, divide or spelling. 

Figure. 9 illustrates a flow diagram of the "play" mode of 

30 the multi-modal device 10. A signal, representative of the 

"play" mode is entered in connector "D" represented by the t 
block 320 and passed to the *play" mode' represented" by block " " ' 
328. From block 328 the signal is directed to block 404 where 
a decision is made by the operator to play music or a game. 

35 once the proper selection of the "play* mode has been made, a 
signal representative of the selection is passed to block 406. 
At this time the operator determines whether to reset the 
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process or continue on. If the process is reset because the 
operator does not wish to proceed, a signal is generated via 
block 406 and directed to connector "A*, represented by the 
block 310 . If the process was not reset, a signal is advanced 
5 to block 408 where the signal is compared to determine if a 

valid selection was made. If the input signal is determined: to 
be incorrect, the signal representing such a determination is 
supplied block 410 where a signal representing an error message 
is produced and fed connector "D", represented by the block 

10 320. If the signal received by the block 408 represents a 

valid selection, a signal is generated and directed to either 
block 412 via connector "N" represented by the block 414 or to 
block 416 via connector "O* represented by the block 420. 
Connector "N" is used for music selection as shown by block 

15 412; and connector n O" is used for game selection as shown by 
block 416. If the operator had selected the music mode, the 
operator would have the option to either continue or "reset" by 
activation of a reset mechanism represented by block 420 so as 
to provide a signal to connector "D", represented by the block 

20 320. If "reset" was not necessary a signal would be passed 
from block 420 to a "play music" block 422, whereupon the 
multi-modal device 10 would play music. If the selection had 
been made to play a game, a signal would proceed from connector 
"0", represented by block 420, to the game block 416, whereupon 

25 the operator could make the selection to either continue or 
reset by activa*ion*of ^ T^serwec^ 
block 424 so as to provide a signal to connector "D" 
represented by the block 320 . If the "reset" had not been 
selected a signal would be passed from block 424 to a "play 

30 game" block 426 whereupon the multi-modal device 10 would play 
a game as preselected. - In either of the above selections the 
device 10 would either play music or play a game and would 
continue until the completion of the music or the game, 
whereupon it could be "reset" and go to connector "D" 

35 represented, by block 320, or continue to play other music, 
tunes or games.. 

Figure 10 is a flow diagram showing the "show me" function 
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of multi-modal device 10. The "show me* function is ent red 
via a signal through connector "E", represented by block 322 
whereupon the signal proceeds to the "show me" block 330. The 
operator has the possibility of selecting one of several 
5 possible functions represented by block 434, such as a solution 
function, a score function, or a mathematical tables function. 
Upon making a selection of one of the three above functions a 
signal is generated and the operator must determine whether to 
activate a reset mechanism represented by block 436 or continue 

10 on. If the reset mechanism is activated a signal is directed 
to connector n A n , represented by the numeral 310. If reset, is 
not necessary, a signal from the block 436 is passed to a valid 
selection block 438. In the event an invalid selection is made 
a signal is directed to an error message block 440 whereupon a 

15 signal representing an error message would be produced and 
passed to connector "E" represented by the block 322. If a 
valid selection is determined by the block 438 the signal is 
passed to either a solutions block 442 via connector "P" 
represented by the block 444, score block 446 via connector "Q" 

20 represented by the block 448, or tables block 450 via connector 
"R" represented by the block 452. For example, if the solution 
function had been selected and the input signal has been 
determined to be a valid signal, the signal would be passed 
through the connector "P" r i.e. block 444 and from there to the 

25 solution block 442. The signal would then proceed to a show 

sol ut ion block -454. -where -a- -cesponse-would ~be~generateed^ to^shew ~ 
the solution to the problem that was being worked on. Upon 
completion of the "show solution" function the process would 
proceed to connector "E" represented by the block 322. 

30 similarly, if the "score" function had been chosen the signal 
would continue from connector "Q", represented by block 448 to 
. the "score" block "446. At that time, the signal would- continue 
to the show score block 456. Upon completion of the score 
being shown by the multi-modal device 10 a signal continues to 

35 connector "E" represented by block 322 for further processing. 
In like manner if the "tables" block 450 has been selected, a 
signal would continue to connector "R", represented by the 
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block 452 and continue onto the "tables" block 450. The signal 
would then pass to block 458 for selection of a proper 
preprogramed table. A signal representing the proper table 
would be transmitted to the show table block 460 whereupon the 
5 table would be transformed into a form suitable for use and 
inspection by the operator. Upon completion of the tables 
being shown, a signal would be transmitted to connector "E", 
represented by block 322 for the continuation of processing . 
The general operation of the multi- modal education and 

10 entertainment device 10 will be described hereinbelow. When 
power is turned on, the multi-modal device 10, through its 
stored program shows a message on the display prompting the 
user to choose one of the control functions. For instance, in 
one form, the following may be the options - define problems, 

15 solve problems, spell, play music or play games. If any choice 
other than those shown above is made, the program provides an 
error feedback and re-prompts the user for the correct choice. 

Assume that "solve* option is selected to solve problems 
previously defined. The program prompts the user to select one 

20 of the following: add, subtract, multiply, divide or spell. 
Contacting any other pad location prompts an error message. 

Now assume that the stylus 18, a non-cognitive element, is 
moved across the two half pads- of the "add" control input pad, - 
thereby selecting the add function. The program now prompts 

25 the user for the first "add." problem to be solved. The subject 

first writes, the problem .toJae.Jsalyed .on^the^ relational ,pads M >i.n. y »^^~. g 
the "add" relational section, such as data input pads 46 of the 
relational section 56. Refering to Figure 2B, this is done as 
follows: in the single entry mode, the stylus 18 is first used 

30 to mark the digit on the appropriate relational pad, thereby 
indicating the pad location being marked on, followed 
immediately by moving the stylus 18 in. an electrically 
contacting position across the two half pads of the appropriate 
cognition pad (data input) 38 in the non-relational section 44. 

35 This procedure informs the logic unit. 82 what digit, (value) was 
entered in the previous operation. In the multiple entry mode, 
several pads in the "add" relational section are written on; 
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then the stylus 18 is moved across the two halves of the pads 
of the appropriate digit(s) in the data input pad 38 in the 
non-relational section 44, the same number of times and the 
same sequence as writing was done in the earlier operation. 
5 This process is continued until the problem is solved. At this 
time the non-relational control input pad "enter* is contacted 
across the two halves, thereby informing the logic unit 82 that 
the subject's solution is ready to be scored. The logic unit 
82, through its stored program(s), solves the problem' and 

10 checks the solution against what the subject entered. If it is 
correct, the program adds 1 to the good score for "add" & adds 
1 to the total add problems solved. This is followed by a 
visual and/or sound display of a "good" message feedback. The 
process then presents the next "add" problem for solution as 

15 before. However, if the solution was in error, a visual and/or 
sound error message is displayed, and the same problem is 
presented for retry. However, the program adds 1 to the total 
attempted for add problems thereby lowering the subject's 
earned score (percentage) for "add". 

20 The subject may do one or more of the following* 

1. By selecting the skip control function, request that 
the present problem be skipped and be presented with 
the next problem. 

2. By selecting the "score" control function, request 
25 the score to be displayed up to this point. 

3. By selecting the- "prompt" -control- function r=- the — 
program checks the validity of the problem as it 
is being solved; providing a visual and/or sound 
feedback when an error is detected. 

30 4. By activating the "Show Me" pad followed by the 

"Answer" pad, the solution to the problem is 
displayed. Once the "Answer" is selected, the 
process can be such that the subject cannot get a 
score on that problem. 
35 The process herein above described shows problems maybe 

solved by means of the "add" mode as an illustration. The 
process is similar for subtract, multiply and divide. 
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When the "solve" control function is followed by the 
"spell" control function, the logic unit 82 picks the first 
word defined for spelling, and through the voice synthesizer, 
converts it to sound impulses. The subject then spells the 
5 words by writing on the "spelling" pads 54 in the relational 
section 64 followed by moving the writing medium across the two 
halves of the data input pads 38 in the non-relational section 
44 in the same sequence as in which the letters were written on 
the spelling pads. When the spelling has been accomplished, 

10 the "enter" control function is selected. As in the case of 
math, the logic unit 82 checks what was entered against the 
defined word for correctness and scoring. 

The multi-modal device 10 has provisions for defining 
problems through itself, apart from the master computer 12. 

15 This feature gives the parent/guardian of a child theoptioh to 
define problems more suited to the needs of a child. Problem 
definitions are eaily accomplished as follows: 

First the stylus 18 is contactingly moved across the two 
halves of the input pad labelled "define" to select the define 

20 mode. 

Assume that the following two numbers are required to be 
addedr 427 + 608. The stylus 18 is contactually moved, across 
the two halves of the data input pads 38 in the non-relational 
section 44 labeled with the digits and symbols in the same 

25 sequence as the sequence shown. The process is terminated by 
activiting the Renter" -pad^whioh^signals^the^logic^^ 
store the defined problem for future retrieval and use. This 
is very easily and simply accomplished. 

The method of defining other types of problems i.e. 

30 subtract, multiply and divide, is similar to that for the "add" 
mode. These are simply done by writing the problems as" a " 
string with the operator ( +, -, x, -./..) imbedded within the 
string as shown herein above for "add* and entered as d scribed 
for "add". The logic unit 82 stores, the problem defined in its 

35 memory unit 22 for future use. The defining of words for 

spelling is accomplished in a similar manner. However, after 
choosing the "define" control option, the "spell" control 



WO 84/02995 



-28 



PCT/US84/00129 



option may be selected to inform the logic unit 82 that 
spelling words are to be defined. 

In a similar fashion, musical scores may be defined. The 
note to be played followed by the time duration are written as 
5 a string. The define control function is first chosen, 

followed by the "music 1 ' control function. The stylus 18 is 
then moved across the appropriate data input pads 38 in the 
non- relation section 44 in the same sequence as the string 
established. The logic unit 82 then stores these values in its 
3 memory unit 84 for future retrieval for producing the tune. 

The multi-modal device 10 also has provisions for a child 
to play games. This is done by selecting the game option 
followed by the game control functions. The logic unit 82 is 
programmed to prompt one or more subjects through the games. 
These may be, for instance, playing word guessing games, number 
guessing games, tic-tac-toe etc. 

The function of one other set of pads needs to be 
described. These are two vertical sets of pads 36 comprising 
the non-relational section 42. These are provided as a means 
to have feedback for verification purposes. For instance, when 
solving an add problem, activating the pad marked 14/15 
displays the carry digits stored in the memory unit 84 which 
can be visually verified against what is written on the carry 
pads. Any discrepancy noted can then be corrected. Two 
vertical rows are provided because of the fact that only eight 
display characters- - are "displayed-^f'a fUStTSn ~£fie ' embodiment ' 
shown. Activating one of the "left display pads" 36 will 
display up to eight characters/numbers in the same row to the 
right of it; activating one of the "right display pads" 36 will 
display up to eight characters/numbers in the same row to the 
left of it. 

Tests and homework assignments in a classroom situation 
using the present invention are carried out as follows. The 
teacher defines the problems or assignments to be worked on by 
the children in the master computer 12. Through predefined 
communication procedures between the master computer 12 and the 
multimodal devices 10 or 210 connected to it through buss 14, 
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the problems or assignments ate "shipped down 1 * to the 
individual multi-modal devices 10 or 210 where they are stored 
in the memory unit 84 (shown in Figures 3). Each child is then 
able to retrieve and solve the problems as previously defined. 
5 in the event the children are being tested , at the 

expiration of the appropriate length. of time, the teacher can 
"pull in" the test scores from each of the multi-modal devices 
10 or 210 into the master computer 12 for tabulation and 
storage.. 

10 In the event the children are solving home-work problems, 

the problems "shipped down" to the multi-modal devices 10 or 
210 from the master computer 12 are stored in the memory 
portion of the memory unit 84. At the end of the day, the 
children disconnect buss 14 from the multi-modal devices 10 or 

15 210, carry it home and solve the assignments. The next day, 

when the multi-modal devices 10 or 210 are connected to buss 14 
and thereby to the master computer 12, the score for each child 
can be "pulled in" as before. 

The above process reduces the time required by teachers to 

20 do routine chores of grading papers and other time consuming 
book keeping tasks. This will free up time on the part of 
teachers that may be fruitfully used for providing personal 
attention to children that need help* 

It is clear that the present invention is well adapted to 

25 carry out the objects and attain the ends and advantages 

mentioned : as well as- pr¥sin€ly^ 
preferred embodiment of the invention has been described for 
purposes of this disclosure, numerous changes may be made which 
will readily suggest themselves to those skilled in the art and 

30 which are encompassed within the spirit of the invention 
disclosed and as defined in" the following claims. 
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What is claimed is: 

1. An improved multi-modal automated educational and 
entertainment device comprising: 

a housing; 

logic means for processing and comparing pre- 
5 programmed data with input and control data? 

memory means for storing programs and input 
data r the memory means being operably 
connected to the logic means; 

a plurality of first input means for pro- 
10 viding. control signals and predefined data 

signals to the logic means r the first input 
means forming a plurality of non-relational 
sections within predefined portions of the 
housing; and 

15 a plurality of second input means for 

providing location indication signals 
representing the location of variable data, 
the second input means forming at least one 
relational section within a predefined portion 

20 of the housing ♦ 

2. The improved multi-modal automated educational and 
entertainment device of claim 1 further comprising-a stylus 
means for selectively activating the first and second input 
means. 

m 

3* The improved multi-modal automated educational and 
entertainment device of claim 1 -further comprising: 
reinforcement means operably connected to 

the logic means for receiving a signal 
5 representative of the results of the 

comparison of the preprogrammed data of the 
logic means and memory means with the control 
signal s, location indication signals and input 
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data supplied the logic. means via activation 
of the first and second input means, and 
producing a reinforcement signal. 



4. The improved multi-modal automated educational and 
entertainment device of claim 2 wherein the logic means is 
connected to the first and second input means via a first 
bi-directional buss and the logic means is connected to the 

5 stylus means via a second bi-directional buss* 

5. The improved multi-modal automated educational and 
entertainment device of claim 3 wherein the reinforcement means 
comprises a voice unit electrically connected to the logic 
means and adapted to receive a signal, from the logic means so 

5 that the voice unit can convert the signal into an audio sound 
for positive and negative reinforcement to an operator of the 



6. The improved multi-modal automated educational and 
entertainment device of claim 5 wherein the voice unit further 
comprises a voice synthesizer and a receiver , the voice 
synthesizer operably connected to the receiver for providing 

5 audio messages to the operator of the device. 

7. The improved multi-modal automated educational and 
-entertainment device o£->clalm^ 

comprises a tone generator operably connected to the receiver 
for providing audio signals to the operator of the device. 

8. The improved multi-modal automated educational and 
entertainment device- of claim 3 further comprising visual 
display means electrically connected" to ther logic Tneains-f or 
providing a visual readout of signals transmitted to the visual 

5 display means from the logic means. 

9; The improved multi-modal automated educational and 
entertainment device of claim 8 wherein the visual display 



device. 
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means comprises a plurality of lights and alphanumeric LEDs or 
LCDs. 

10* The improved multi-modal automated educational and 
entertainment device of claim 1 wherein the logic means, is 
provided with an integrated input/output port f the input/output 
port electrically connectable to a master computer for 
5 transmission and receipt of control and input/output data to 
and from the master computer to the logic means of the device, 

11. The improved multi-modal automated educational and 
entertainment device of claim 10 wherein a plurality of the 
devices are operably connected to a master computer , each of 
the devices adapted to transmit and receive control and 

5 input/output data to and from the master computer independently 
of each other. 

12. The improved multi-modal automated educational and 
entertainment device of claim 4 wherein the first and second 
input means each comprises a plurality of input pads r each of 
the pads electrically connected to the logic means such that 

5 upon activation of the pads signals are .directed to the logic 
means representing location indication, input and control data - . 

13. The improved multi-modal automated educational and 
entertainment device of.,cla.im,12..wh^rein jaach~input-.pad.is -~->. 
provided with a body portion having an upper surface, and an 
opposed lower surface, the body portion being constructed of an 

5 electrically conductive material such that upon contacting the 
upper surface of the body portion with the electrically 
conducting stylus, electrical signals are transmitted to the 
logic means* 

14. The improved multi-modal automated educational and 
entertainment device of claim 13 wherein each input pad 
comprises; 

a body portion constructed of an electri- 
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cally conductive material, the body portion 
having an upper surface and an opposed lower 
surface, the body portion further 
characterized as having a first half and a 
second half; 

>0 an insulating medium disposed between the 

first and second halves of the body portion 
such that the first half is electrically 
insulated from the second half; 
a first conductive lead electrically con- 

-5 necting the first half of the body portion to 

the logic means; and 
a second electrically conductive lead 

electrically connecting the second half of the 
body portion to the logic means. 

15 . The improved multi-modal ■ automated educational and 
entertainment device of claim 14 wherein the input pads of the 
second input means forming the relational section of the device 
are further characterized as having a scribing surface formed 

5 on the upper surface of the first and second halves of the body 
portion such that visible markings may be made thereon with the 
stylus and may be easily- erased 1 - for reuse-.- - 

16. The improved multi- modal automated educational and 
entertainment .device o&-*oladm vl3^erein-.4^,iinput^ads^v thfi^ 
second input means forming the relational section of the device 
are further characterized as having a scribing surface formed 

5 on the upper surface of the body portion such that visible, 
marking may be made thereon with the stylus and may be easily 
erased for reuse. 

17. The improved multi-modal automated educational and 
entertainment device of claim 1, wherein each of the first and 
second input means comprises: 

at least switch means for providing an 
5 input signal to the logic means, the switch 
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means being activated by the stylus means and 
the input signal produced thereby being 
indicative of the specific switch means being 
activated. 

18. The improved multi-modal automated educational and 
entertainment device of claim. 1, wherein each of the first and 
second input means comprises: 

at least one electrically conductive means 
5 for providing an input signal to the logic 

means, the electrically conductive means being 
activated by the stylus being brought into 
electrical conduction with a selected one of 
the first and second input means and the input 
10 signal produced thereby being indicative of 

the specific electrically conductive means 
being activated. 

19. The improved multi-modal automated educational and 
entertainment device of claim 14 wherein the first half of the 
pads of the second input means forming the relational section 
are precoded, the first halves of the pads bearing the same 

5 code being electrically connected together to form rows of 
pads? and wherein the second half of the pads of the second 
input means forming the relational section are preceded, the 
second halves of the pads bearing the" same" code 'being" 
electrically connected together to form columns of pads such 

10 that upon contacting the first and second halves of a pad with 
the stylus, input impulse signals are transmitted to the logic 
means representative of the row and column location of the pad 
such that the logic means through stored programs can identify 

- - the specif ic pad locatibnT ." " 

20. The improved multi-modal automated educational and 
entertainment device of claim 14 wherein the first half of the 
pads of the first input means forming the non-relational 
section are precoded, the first halves of the pads bearing the 
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5 sane code being electrically connected together to form rows of 
pads? and wherein the second half of the pads of the first 
input means forming the non-relational section are precoded, 
the second halves of the pads bearing the same code being 
electrically connected together to form columns of pads such 
10 that upon contacting the first and second halves of a pad with 
the stylus, input impulse signals are transmitted to the logic 
means representative of the row and column location of the pad 
such that the logic means, through stored programs can identify 
the specific pad location. 

21. The improved multirmodal automated educational and 
entertainment device of claim 1 further comprising scratch pad 
means disposed on the upper surface of the housing for allowing 
the operater of the device to scribe thereon, information 

5 useful in the operation of the device. 

22. The improved multi-modal automated educational and 
entertainment device of claim 2 wherein the stylus means is 
used to marie indicia on the second input means, and wherein the 
stylus means is further characterized as comprising: 

5 cognitive means for providing indicia 

indicating signals to the logic means for 
deciphering the marked indicia. 

23. The improved raulti'-modai' -automated educationalr-and ~ - 
entertainment device of claim 1 wherein at least one of the 
second input means is characterized as comprising: 

cognitive means for providing indicia 
5 indicating signals responsive to indicia 

marked on the second input means, the indicia 
indicating signals provided to the logic means 
for deciphering the marked indicia. 



24. The improved multi-modal automated educational and 
entertainment device of claim 1 further comprising: 

power means for activating the logic means 
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into an operable mode* . 

25. The improved multi-modal automated educational and 
entertainment device of claim 1 further comprising; 

overlay means positionable on the housing 

for defining a selected portion of the second 
5 input means* 

26. The improved multi-modal automated educational and 
entertainment device of claim 25 further comprising: 
overlay means positionable on the housing 

for defining a selected portion of the second 
5 input means f the overlay means having at least 

one opening formed therein for exposing the 
selected portion of the second input means, 
positioning means for positioning the 

overlay means in a stable position on the 
10 housing* 

27. The improved multi-modal automated educational and 
entertainment device of claim 2 wherein the stylus means 
comprises: 

a stylus having a body portion and a tip portion 
5 supported thereby, the tip portion being formed of a 

substance capable of making visual marking on the 
second .ii^t?TiiBfanB^^^^^-- fc; "- 



28 ♦ The improved multi-modal automated educational and 
entertainment device of claim 2 wherein the stylus means 
comprises a stylus having an electrically conductive tip. 

29. The improved multi-modal automated educational and 
entertainment device of claim 2 wherein the stylus means 
comprises a stylus having an electrically conductive tip, 
elecltrically connected to the logic means. 



30 • The improved multi-modal automated educational and 
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entertainment device of claim 2 wherein the stylus means 
comprises : 

a cognitive stylus capable of producing 
5 indicia indicating signals to the logic means 

via electrically conductive lines or RF 
signals. 

31. An improved multi-modal automated educational and 
entertainment device of claim 1 wherein the second input means 
are positioned on the upper surface of the housing in a 
relational position so as to relate to the positions of digits 

5 in a mathematical problem. 

32. An improved multi-modal automated educational and 
entertainment device of claim 1 wherein the second input means 
are positioned on the upper surface of the housing in a 
relational position so as to relate to the positions of letters 

5 in a spelling word/problem. 

33. An improved process for information input using 
electromechanical, storage, control, comparison, retrieval and 
output devices comprising the steps of: 

inputting a first input signal'- to a logic 
5 means representing control and predefined data 

information; and 
inputting, a second* input^ignaivtsi. thet logical rsawsac 
means representing location indication 
information, such that the logic. means can 
10 proceed in accordance with a predesignated 

program of operations. 

34. The process of claim 33 further characterized-as 
comprising: 

producing the first and second input 

signals in a predefined sequence of control 
5 and data input activation. 
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35. The process of claim 33 wher in the logic means is 
operably connected to a memory means having at least one 
storage device therewith, and wherein the second input signal 
has alphanumeric information having mathematical control 

5 indicia into the storage device of the memory means* 

36. The process of claim 33 wherein the logic means is 
operably connected to a master computer and the process further 
comprises: 

inputting alphanumeric information 
5 to a plurality of storage devices within a 

memory means ; 
inputting control information to a plurality 

of storage devices within the memory means; 
inputting programs , data or control infor- 
10 mation- from a master computer; and 

selecting and processing of one or more 

major functions contained within the program* 

37. The improved multi-modal automated educational and 
entertainment process of claim 33 , wherein the major function 
is a "DEFINE* function, and the process is further 
characterized as comprising: 

5 selecting and processing a predetermined 

subf unction within the w def ine" function, 
wherein.* the-^proscessin#dLs-^ 
means of the logic means being operably 
connected to the memory means and processed in 
10 accordance with the predetermined program. 

38. The improved multi-modal automated educational and 
entertainment process of claim 33, wherein the major function 
is a "SOLVE 1 * function, and the process is further characterized 
as comprising: 

5 selecting and processing a predetermined 

subf unction within the n solve* function, 
wherein the processing is accomplished by 
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means of the logic means being operably 
connected to the memory means and processed in 
10 accordance with the predetermined program. 

39.. The improved multi-modal automated educational and 
entertainment process of claim 33 , wherein the major function 
is a W PIAY" function, and the process is further characterized 
as comprising: 

5 selecting and processing a predetermined 

subfunction within the "play" function, 
wherein the processing is accomplished by 
means of the logic means, being operably 
connected to the memory means and processed in 
Iq accordance with the predetermined program. 

40. The improved multi-modal automated educational and 
entertainment process of claim 33, wherein the major function 
is a "SHOW ME" function, and the process is further 
characterized as comprising: 

5 selecting and processing a predetermined 

subfunction with the "show me" function, 
wherein the processing is accomplished by 
means of the logic means, being operably 
connected to the memory means and processed in 
10 accordance with the predetermined program. 

41. The process of claim 33 wherein the production of 
second input signals through a non-cognitive process is 
characterized by a stylus means being brought into indicia 
marking engagement with a relational input pad means, 

5 activating the pad location and providing input signals to* the 
logic means as to the location where marking is accomplished, 
the marking being indicative of the indicia of the data being 
input, followed by the stylus being brought into activating 
engagement with a non-relational data input pad means 
10 corresponding to the previously marked indicia, thereby 

providing input to the logic means corresponding to the indicia 
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marked on the relational pad. 

42. The process of claim 41 wherein the multiple indicia 
marking engagement of the relational pad means is followed by 
the same number of indicia marking engagement of the 
non-relational data input pad means, the activation of the 

5 non-relational data input pad corresponding to the marked 

indicia on the relational input pad means in the same sequence* 

43. The process of claim 33 wherein the second input 
signal is produced by a cognitive process comprising engaging a 
stylus, means in indicia marking engagement with a relational 
data input pad means, marking a desired indicia, providing 

5 input to the logic means corresponding to the pad location 
where marking is accomplished, transmitting indicia signals 
therein produced to the logic means r activating an activate 
control pad which directs the logic means to decipher the 
signals produced by the stylus means so that the logic means 

10 deciphers and provides feedback information on the indicia 

recognized, the stylus being capable of activating engagement 
with a non-relational data input pad means corresponding to the 
previously marked indicia to provide a correct data input into 
the logic means when non-recognition of the marked indicia 

15 occurs. 
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